Electronic device and method of controlling the same

ABSTRACT

An example method of controlling an electronic device worn by a user includes constructing a user model by training a content feature according to response characteristics of an eye of a user who wears the electronic device, and in response to a content feature stored in the user model being detected from reproduced content during content reproduction, processing the reproduced content based on response characteristics of the eye of the user corresponding to the detected content feature.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority from Korean PatentApplication Nos. 10-2016-0177343 and 10-2017-0134610, filed on Dec. 23,2016 and Oct. 17, 2017, respectively, in the Korean IntellectualProperty Office, the contents of each of which are incorporated byreference herein in their entirety.

BACKGROUND Technical Field

The present disclosure generally relates to an electronic device and amethod of controlling the same, and more particularly, to an electronicdevice and a method of controlling the same for learning (or training) acontent feature according to response characteristics of a user eye tocontrol reproduction of content reproduced according to the contentfeature.

In addition, the present disclosure relates to an artificialintelligence (AI) system and an application technology thereof forimitating a function of recognition and determination of a human brainusing a machine learning algorithm such as deep learning.

Description of Related Art

Recently, electronic devices (e.g., a head mounted device (HMD)) worn bya user to provide virtual reality have attracted much attention. Forexample, when wearing an HMD, a user may view and enjoy a realisticstereoscopic view in a virtual world that is definitely different fromreality. In addition, a user may enjoy an existing two-dimensional(2D)-based game as a more realistic game with a view of 360 degrees.Thus, virtual reality (VR) contents have been introduced starting fromgame contents and are expected to be used in various fields such asdistance education and medical treatment via a service of sharingexperience of virtual reality with a remote site.

In the case of such a HMD, user eyes and an electronic device are veryclose to each other and, thus, a visual stimulus applied to the usereyes according to a change in specific color and specific brightness maybe very high. Thereby, fatigue of user eyes is gradually increased and,in a more serious case, there is the possibility that an illness iscaused in the user eye.

When a user has a phobia about a specific object (e.g., a knife and asharp object), if the specific object appears in an image provided by anHMD, the user may go through the inconvenience of viewing an image usingthe HMD that triggers this phobia.

Recently, artificial intelligence (AI) systems have also been introducedin the image processing field.

An AI system is a computer system for realizing intelligence of a humanlevel and is a system that becomes more intelligent through autonomouslearning and determination of a machine different from an existingrule-based smart system. As the AI system is further used, a recognitionrate is enhanced and user preference is more accurately understood.Accordingly, existing rule-based smart systems have been graduallyreplaced with deep learning-based AI systems.

Artificial intelligence (AI) technology is configured with machinelearning (deep learning) and element technologies using machinelearning.

Machine learning is algorithm technology for autonomouslyclassifying/learning a feature of input data, and element technologiesare for imitating a function such as recognition and determination ofhuman brain using a machine learning algorithm such as deep learning.These are configured for technological fields such as linguisticunderstanding, visual understanding, inference/prediction, knowledgerepresentation, and motion control.

Various fields to which AI technologies are applied are now described.Linguistic understanding is technology for recognizing/processing humanlanguages/characters and includes natural language processing, machinetranslation, dialogue system, question and answer, voicerecognition/synthesis, and so on. Visual understanding is technology forrecognizing and processing an object like human visual sense andincludes object recognition, object tracking, image search, humanrecognition, scene understanding, space understanding, imageimprovement, and so on. Inference and prediction is technology fordetermining information to acquire logical inference and prediction andknowledge/possibility-based inference, optimization prediction, apreference-based plan, recommendation, and so on. Knowledgerepresentation is technology for automatically processing humanexperience information to knowledge data and includes knowledgeconstruction (data generation/classification), knowledge management(data use), and so on. Motion control is technology for controllingautonomous driving of a vehicle and motion of a robot and includesmotion control (navigation, collision, and driving), manipulationcontrol (behavior control), and so on.

SUMMARY

Example embodiments of the present disclosure may overcome the abovedisadvantages and other disadvantages not described above. Also, thepresent disclosure is not required to overcome the disadvantagesdescribed above, and an example embodiment of the present disclosure maynot overcome any of the problems described above.

The present disclosure provides an example electronic device and anexample method of controlling the same for learning a content featureaccording to response characteristics of a user eye to construct a usermodel and for reproducing content using the constructed user model.

According to an aspect of the present disclosure, an example method ofcontrolling an electronic device worn by a user to provide an imageincludes learning a content feature according to responsecharacteristics of an eye of a user who wears the electronic device andconstructing a user model, and, in response to a content feature storedin the user model being detected from the reproduced content duringcontent reproduction, processing the reproduced content based onresponse characteristics of the user eye corresponding to the detectedcontent feature.

According to another aspect of the present disclosure, an exampleelectronic device worn by a user to provide an image includes a displayconfigured to display content, an image capture device (e.g., camera)configured to photograph a user eye, a memory configured to learn acontent feature according to response characteristics of an eye of auser who wears the electronic device and to store a constructed usermodel, and a processor configured to, in response to a content featurestored in the user model being detected from the reproduced contentduring content reproduction, process the reproduced content based onresponse characteristics of the user eye corresponding to the detectedcontent feature.

According to the diverse example embodiments of the present disclosure,an image of a content feature expressed as a user negative response maybe processed and, thus, the user may view the image more convenientlywithout stimulation. In addition, disease information of a user eye maybe observed and, thus, information on an abnormal state of the user eyemay be provided.

Additional and/or other aspects and advantages of the present disclosurewill be set forth in part in the description which follows and, in part,will be obvious from the description, or may be learned by practice ofthe present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects, features, and attendant advantages ofthe present disclosure will be more apparent and readily understood fromthe following detailed description of certain example embodiments of thepresent disclosure, taken in conjunction with the accompanying drawings,in which like reference numerals refer to like elements, and wherein:

FIGS. 1A and 1B are schematic block diagrams showing a configuration ofan electronic device according to an example embodiment of the presentdisclosure;

FIG. 2 is a block diagram showing a configuration of an electronicdevice in detail according to an example embodiment of the presentdisclosure;

FIG. 3 is a block diagram showing a configuration of a processorincluded in an electronic device according to an example embodiment ofthe present disclosure;

FIG. 4A is a block diagram showing a configuration of a data learner anda content controller included in an electronic device according to anexample embodiment of the present disclosure;

FIG. 4B is a diagram for explanation of an example in which anelectronic device and a server are operatively associated to learn andrecognize data according to an example embodiment of the presentdisclosure;

FIG. 5 is a diagram for explanation of a data learner according to anexample embodiment of the present disclosure;

FIG. 6 is a diagram for explanation of a method of processing content bya content controller according to an example embodiment of the presentdisclosure;

FIGS. 7A and 7B are diagrams for showing a user interface (UI) forcorrecting a response feature of a user eye and a content featureaccording to an example embodiment of the present disclosure;

FIGS. 8A and 8B are diagrams for explanation of an example in which animage is processed with respect to an object to which a user hasnegative emotion according to an example embodiment of the presentdisclosure;

FIG. 9 is a diagram for explanation of an example of determining diseaseinformation of a user eye according to an example embodiment of thepresent disclosure;

FIG. 10 is a diagram showing a user interface (UI) for guiding diseaseinformation of a user eye according to an example embodiment of thepresent disclosure;

FIG. 11 is a flowchart of a method of controlling an electronic deviceaccording to an example embodiment of the present disclosure;

FIG. 12 is a diagram for explanation of a system including a portableterminal and an electronic device according to an example embodiment ofthe present disclosure;

FIG. 13 is a flowchart for explanation of a case of estimating a contentfeature when an electronic device includes a first processor and asecond processor according to an example embodiment of the presentdisclosure;

FIG. 14 is a flowchart for explanation of a case of estimating a contentfeature using a server by an electronic device according to an exampleembodiment of the present disclosure; and

FIG. 15 is a flowchart for explanation of a case of estimating anabnormal state related to an eye using a server by an electronic deviceaccording to an example embodiment of the present disclosure.

DETAILED DESCRIPTION

The present disclosure will now be described more fully with referenceto the accompanying drawings, in which example embodiments of thepresent disclosure are shown. In the following description of thepresent disclosure, a detailed description of known functions andconfigurations will be omitted when it may make the subject matter ofthe present disclosure unclear. The terms used in the specification aredefined in consideration of functions used in the present disclosure,and can be changed according to the intent or conventionally usedmethods of clients, operators, and users. Accordingly, definitions ofthe terms should be understood on the basis of the entire description ofthe present disclosure.

It will be understood that, although the terms first, second, third etc.may be used herein to describe various elements, these elements shouldnot be limited by these terms. These terms are only used to distinguishone element from another element. For example, a first element may betermed a second element and a second element may be termed a firstelement without departing from the teachings of the present disclosure.As used herein, the term “and/or” includes any and all combinations ofone or more of the associated listed items.

The terminology used herein is for the purpose of describing particularexample embodiments only and is not intended to be limiting of exampleembodiments. As used herein, the singular forms “a,” “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises” and/or “comprising” used herein specify the presence ofstated features, integers, steps, operations, members, components,and/or groups thereof, but do not preclude the presence or addition ofone or more other features, integers, steps, operations, members,components, and/or groups thereof.

In example embodiments of the present disclosure, terms such as “unit”,“module”, etc. used in the specification may refer to units forprocessing at least one function or operation, which may be implementedby hardware, software, or a combination thereof. In addition, aplurality of ‘modules’ or a plurality of ‘units’ may be integrated intoat least one module to be embodied as at least one processor except fora ‘module’ or a ‘unit’ that needs to be embodied as specific hardware.

Hereinafter, the present disclosure will be described in detail withreference to the accompanying drawings. FIG. 1A is a schematic blockdiagram showing a configuration of an electronic device 100 according toan example embodiment of the present disclosure. As shown in FIG. 1A,the electronic device 100 may include a display 110, an image capturedevice (e.g., camera) 120, a processor 130, and a memory 140. In thiscase, as shown in FIG. 1B, the electronic device 100 may be a head-updisplay (HUD) that is worn by a user head to provide an image, but thisis merely an example embodiment of the present disclosure and theelectronic device 100 may be implemented as other types of electronicdevices (e.g., a smartphone, a tablet personal computer (PC), a notebookPC, and the like).

The display 110 may display an image. In particular, the display 110 maydisplay an image(s) acquired from various sources as 3D image(s). Inthis case, when a user wears the electronic device 100, the display 110may be positioned on an internal surface of the electronic device 100,which is viewed by a user's eyes. Accordingly, the display 110 mayprovide a stereoscopic image with a high sense of immersion to a userwho wears the electronic device 100.

The image capture device (e.g., camera) 120 may photograph a user'seyes. In particular, the image capture device 120 may be positioned onthe internal surface of the electronic device 100 to photograph the usereyes.

The memory 140 may store various data and programs for control of theelectronic device 100. In particular, the memory 140 may store a usermodel constructed by learning a content feature according to responsecharacteristics of the user's eyes photographed by the image capturedevice 120.

The processor 130 may control an overall operation of the electronicdevice 100. Detecting the content feature stored in the user model fromreproduced content during reproduction of the content, the processor 130may process the reproduced content based on the response characteristicsof the eyes of a user corresponding to the detected content feature.

In detail, the processor 130 may learn a content feature according tothe response characteristics of the user's eyes photographed by theimage capture device 120 to construct the user model. In this case, theresponse characteristics of the user's eyes may be a feature of a shape,size, and color change of a user eye or reaction velocity of a user eyeand may include without limitation eye blink, eye closure, eye frown,pupil dilation, pupil contraction, and the like.

In more detail, the processor 130 may capture an image containing aneye(s) of a user who wears the electronic device 100 through the imagecapture device 120 while learned content is reproduced. Upon detectingpredetermined response characteristics of the user's eye(s) included inthe captured image, the processor 130 may acquire a content featureincluded on a content frame within a predetermined content section froma time point at which the predetermined response characteristics aredetected. The processor 130 may learn predetermined responsecharacteristics and a content feature to construct the user model. Inthis case, the content feature may be at least one of an object (e.g., aknife and a ghost) included in the content frame within thepredetermined content section from a time point at which thepredetermined response characteristics are detected, a brightness changein the content frame, or a color change in the content frame. Theconstructed user model may store the predetermined responsecharacteristics corresponding to the content feature.

The processor 130 may process an image of content based on theconstructed user model.

In detail, the processor 130 may be configured to analyze reproducedcontent during content reproduction to determine whether a contentfeature stored in the user model is present. When a first contentfeature stored in the user model is present in the reproduced content,the processor 130 may process of an image corresponding to a contentframe including the first content feature based on the first contentfeature and response characteristics corresponding to the first contentfeature.

According to an example embodiment of the present disclosure, when thefirst content feature is a specific object and response characteristicsof the specific object are a negative response (e.g., eye blink, eyeclosure, and/or eye frown), the processor 130 may perform filteringprocessing on the object included in the content or may performsmoothing processing. According to another example embodiment of thepresent disclosure, when the first content feature is a specificbrightness change and a user response to the specific brightness changeis negative, the processor 130 may adjust a variation amount of thespecific brightness change. According to another example embodiment ofthe present disclosure, when the first content feature is a specificcolor change and a user response to the specific color change isnegative, the processor 130 may adjust a saturation value of thespecific color change.

Upon determining that a user response to a second content feature of thelearned user model is positive, the processor 130 may acquire a keywordof a content feature and may control the display 110 to provide a listincluding a determined recommendation content based on the acquiredkeyword in response to a user command.

The processor 130 may cumulatively store the captured image including auser's eye(s) in the memory 140 and may analyze the stored imagesincluding the user's eye(s) to determine an abnormal state of the user'seye(s). Upon determining that the user's eye(s) is abnormal, theprocessor 130 may control the display 110 to provide information on theabnormal state.

FIG. 2 is a block diagram showing a configuration of the electronicdevice 100 in detail according to an example embodiment of the presentdisclosure. As shown in FIG. 2, the electronic device 100 may includethe display 110, the image capture device 120, the memory 140, acommunicator (e.g., including communication circuitry) 150, an imageprocessor 160, an audio output device (e.g., including audio outputcircuitry) 170, an input device (e.g., including input circuitry) 180, adetector 190, and the processor 130.

The display 110 may display various image content, information, a userinterface (UI), etc. provided by the electronic device 100. For example,the display 110 may display various execution images of a navigationapplication.

For example, the display 110 may display an image(s) acquired fromvarious sources as 3D image(s). In particular, the display 110 maydisplay a left-eye image through a display corresponding to a left eyeof a user and may display a right-eye image through a displaycorresponding to a right eye of the user to display a 3D image.

In this case, when the user wears the electronic device 100, the display110 may be positioned on the internal surface of the electronic device100, which is viewed by the user. Accordingly, the display 110 mayprovide a stereoscopic image with a high sense of immersion to a userwho wears the electronic device 100.

When the user wears the electronic device 100, the image capture device(e.g., including a camera) 120 may be positioned on the internal surfaceof the electronic device 100 to photograph a user's eye(s). In thiscase, the image capture device 120 may be a general-purpose camera, butthis is merely an example embodiment and the image capture device 120may also be implemented with or as an infrared camera or the like.

The memory 140 may store various programs and data required for anoperation of the electronic device 100. The memory 140 may beimplemented as a non-volatile memory, a volatile memory, a flash memory,a hard disk drive (HDD), a solid state drive (SSD), and/or the like. Thememory 140 may be accessed by the processor 130 and the processor 130may read/record/correct/delete/update data. In the present disclosure,the term ‘memory’ may include the memory 140, ROM (not shown) and/or RAM(not shown) in the processor 130, and/or a memory card (not shown)(e.g., a micro SD card and a memory stick) installed in the electronicdevice 100.

The memory 140 may store a program, data, and the like for configuringvarious images to be displayed on a display region of the display 110.The memory 140 may store the user model for storing a content featurecorresponding to response characteristics of a user's eye(s) accordingto an example embodiment of the present disclosure.

The communicator (e.g., including communication circuitry) 150 maycommunicate with various types of external devices using various typesof communication methods. The communicator 150 may include at least oneof a WiFi chip, a Bluetooth chip, a wireless communication chip, or anNFC chip. The processor 130 may communicate with an external server orvarious external devices using the communicator 150.

In particular, the communicator 150 may communicate with an externalelectronic device (e.g., a smartphone, a navigation device, and aserver). For example, the communicator 150 may receive image contentfrom an external electronic device.

The image processor 160 may process an image(s) of image data receivedfrom various sources. The image processor 160 may perform various imageprocessing such as decoding, scaling, noise filtering, frame rateconversion, and resolution conversion on the image data.

The audio output device (e.g., including a speaker and associated audiooutput circuitry) 170 may output various alarms or voice messages aswell as various audio data processed by an audio processing module.

The input device (e.g., including input circuitry) 180 may accept inputof a user command for manipulating the electronic device 100 and maytransmit information on the user command to the processor 130. Inparticular, the input device 180 may be a button (e.g., a physicalbutton, an optical key, or a keypad) included in the electronic device100, but this is merely an example embodiment and the input device 180may be implemented using additional or alternative input devices.

For example, the input device 180 may be implemented as a touch panel, a(digital) pen sensor, etc. for detecting a user touch. The touch panelmay use at least one of, for example, a capacitive method, a resistivemethod, an infrared method, or an ultrasonic method. The touch panel mayfurther include a tactile layer to provide a tactile response to a user.The (digital) sensor may be, for example, a portion of a touch panel ormay include a separate element.

As noted, the input device 180 may be implemented as a button, touchpanel, a pen sensor, or the like but these are merely exampleembodiments and the input device 180 may be implemented as variouselectronic devices such as a microphone for receiving user speed, acamera for photographing a user motion, and a pointing device.

The detector 190 may include various sensors for detecting a state ofthe electronic device 100. In particular, the detector 190 may include aGPS sensor for detecting a position of the electronic device 100, amotion sensor (e.g., an acceleration sensor, a gyro sensor, and anelectromagnetic sensor) for detecting a motion of the electronic device100, a camera sensor for photographing a path of vehicle driving, and soon.

The processor 130 may be electrically connected to various components(e.g., the display 110, the memory 140, and the image capture device120) of the electronic device 100 to control an overall operation andfunction of the electronic device 100. In particular, the processor 130may control an overall operation of the electronic device 100 usingvarious programs stored in the memory 140.

In detail, as shown in FIG. 3, the processor 130 may include a datalearner 131 for generating a user model 300 and a content controller 132for processing content using the user model 300.

The data learner 131 may learn response characteristics of a user'seye(s) for a content feature. In particular, the data learner 131 maylearn a content feature of a response of the user eye based on an imageof the user eye and the learning content. In this case, the data learner131 may learn whether a user expresses specific response characteristicswith respect to a specific content feature to construct the user model300 for storing a content feature according to response characteristicsof the user's eye(s).

The content controller 132 may process reproduced content based on theuser model 300 generated by the data learner 131. In detail, upondetecting a content feature stored in the user model 300, the contentcontroller 132 may process an image(s) of content based on responsecharacteristics of a user's eye(s) corresponding to the detected contentfeature.

At least one of the data learner 131 and the content controller 132 maybe manufactured in the form of at least one hardware chip and may beinstalled in the electronic device 100. For example, at least one of thedata learner 131 and the content controller 132 may be manufactured inthe form of a dedicated hardware chip for artificial intelligence (AI)or may be manufactured as a part of an existing general-purposeprocessor (e.g., a CPU or an application processor) or a graphicdedicated processor (e.g., a GPU) and may be installed in theaforementioned various electronic devices 100.

In this case, the data learner 131 and the content controller 132 may beinstalled in one electronic device 100 or may be installed in separateelectronic devices, respectively. For example, one of the data learner131 and the content controller 132 may be included in an electronicdevice and the other one may be included in an external portableterminal (e.g., a smartphone) or server. The data learner 131 mayprovide model information constructed by the data learner 131 to thecontent controller 132 by wire or wirelessly.

At least one of the data learner 131 and the content controller 132 maybe implemented as a software module. When at least one of the datalearner 131 and the content controller 132 is implemented as a softwaremodule (or a program module including an instruction), the softwaremodule may be stored in a non-transitory computer readable medium (ormedia). In this case, at least one software module may be provided by anoperating system (OS) or may be provided by a predetermined application.Alternatively, a portion of at least one software module may be providedby an operating system (OS) and the remaining portion may be provided bya predetermined application.

Hereinafter, with reference to FIGS. 4A, 4B, 5 and 6, the features ofthe data learner 131 and the content controller 132 will be described inmore detail.

Referring to FIG. 4A, the data learner 131 according to some exampleembodiments of the present disclosure may include a data acquirer 131-1,a pre-processor 131-2, a learning data selector 131-3, a model learner131-4, and a model evaluator 131-5.

However, the present disclosure is not limited thereto. According tovarious example embodiments of the present disclosure, the data learner131 may include all or some of the aforementioned components. Forexample, the data learner 131 may include only the data acquirer 131-1and the model learner 131-4. In addition, according to various exampleembodiments of the present disclosure, the data learner 131 may furtherinclude other components in addition to the aforementioned components.

The data acquirer 131-1 may acquire learning data required to analyze acontent feature according to response characteristics of a user'seye(s). In this case, the data acquirer 131-1 may acquire image data,content data, etc. including a user's eye(s) photographed by the imagecapture device 120 as the learning data. In this case, the content datamay include audio data and metadata as well as the image data containedin the content data.

The pre-processor 131-2 may pre-process the acquired learning data touse the learning data acquired to analyze a content feature according toresponse characteristics of a user's eye(s). The pre-processor 131-2 mayprocess the acquired data in a predetermined format to use the learningdata acquired for learning for analyzing a content feature according toresponse characteristics of a user's eye(s) by the model learner 131-4that will be described below.

The learning data selector 131-3 may select learning data required forlearning from the pre-processed learning data. The selected learningdata may be provided to the model learner 131-4. The learning dataselector 131-3 may select the learning data required for learning fromthe pre-processed learning data according to a predetermined standardfor analysis of a content feature according to response characteristicsof a user's eye(s). For example, the learning data selector 131-3 mayuse, as learning data, only image data including responsecharacteristics of a predetermined eye of a user.

However, the learning data selector 131-3 may select some learning datafrom the pre-processed learning data, but this is merely an exampleembodiment and, thus, the learning data selector 131-3 may select all ofthe pre-processed learning data. The learning data selector 131-3 mayselect learning data prior to a pre-processing operation by thepre-processor 131-2.

The model learner 131-4 may learn a content feature according toresponse characteristics of a user's eye(s) based on the learning datato construct a user model. In this case, the model learner 131-4 mayanalyze the content feature according to the response characteristics ofthe user's eye(s) and may learn an cumulative analysis result toconstruct the user model. In this case, the user model may storeresponse characteristics of a user's eye(s) and a content feature thatare matched with each other. A method of constructing a user model usinglearning data by the model learner 131-4 will be described in moredetail with reference to FIG. 5.

The model learner 131-4 may make the user model, used to analyze acontent feature according to response characteristics of a user'seye(s), learn using learning data. In this case, the user model may be apre-constructed model. For example, the user model may be a model thatis pre-constructed by receiving basic learning data (e.g., generalresponse characteristics of a user's eye(s) and a general contentfeature).

The user model may be constructed in consideration of an applicationfield of data analysis, a purpose of learning, or computer performanceof a device. The user model may be, for example, a model based on aneutral network. For example, a model, such as a deep neural network(DNN), a recurrent neural network (RNN), and a bidirectional recurrentdeep neural network (BRDNN), may be used as the user model, but thepresent disclosure is not limited in this respect.

According to various example embodiments of the present disclosure, whena plurality of pre-constructed user models are present, the modellearner 131-4 may determine a user model with a high relationshipbetween input learning data and basic learning data as a user model as alearning target. In this case, the basic learning data may bepre-classified for each data type and the user model may bepre-constructed for each data type. For example, the basic learning datamay be pre-classified based on various references such as a generator ofthe learning data, a generation time of the learning data, a size of thelearning data, a genre of the learning data, an object type in thelearning data, and so on.

The model learner 131-4 may make the user model learn using a learningalgorithm including, for example, error back-propagation or gradientdescent.

The model learner 131-4 may make the user model learn through, forexample, supervised learning using learning data as an input value. Themodel learner 131-4 may make the user model learn through, for example,unsupervised learning for autonomously learning a type of data requiredto analyze a content feature according to response characteristics of auser's eye(s) without particular supervision to discover a reference foranalysis of a user driving history. The model learner 131-4 may make theuser model to learn through, for example, reinforcement learning usingfeedback about whether a result of a user's driving history according tolearning is accurate.

When the user model is learned, the model learner 131-4 may store thelearned user model. In this case, the model learner 131-4 may store thelearned user model in the memory 140 of the electronic device 100including the content controller 132. Alternatively or additionally, themodel learner 131-4 may store the learned user model in a memory of anexternal portable terminal or server that is connected to the electronicdevice 100 by wire or via a wireless network.

In this case, the memory 140 that stores the learned user model maystore therewith a command or data related to at least one of othercomponents of the electronic device. The memory 140 may store softwareand/or a program. The program may include, for example, kernel,middleware, an application programming interface (API), and/or anapplication program (or “application”), etc.

The model evaluator 131-5 may input evaluation data to the user modeland, when an analysis result output from the evaluation data does notsatisfy a predetermined reference or criteria, the model evaluator 131-5may make the model learner 131-4 re-learn. In this case, the evaluationdata may be predetermined data for evaluation of the user model.

For example, the model evaluator 131-5 may evaluate that the analysisresult does not satisfy a predetermined reference or criteria when thenumber or rate of evaluation data with an inaccurate analysis resultexceeds a predetermined threshold value from analysis results of thelearned user model with respect to the evaluation data. For example,when the predetermined reference or criteria is defined as a rate of 2%,the learned user model outputs an inaccurate analysis result withrespect to evaluation data, the number of which exceeds 20, from a totalof 1000 evaluation data, the model evaluator 131-5 may evaluate that thelearned user model is not appropriate.

When a plurality of learned user models is present, the model evaluator131-5 may evaluate whether each learned user model satisfies apredetermined reference or criteria and may determine a model thatsatisfies the predetermined reference or criteria as a final user model.In this case, when a plurality of models satisfies the predeterminedreference or criteria, the model evaluator 131-5 may determine one or apredetermined number of models as the final user model in descendingorder of an evaluation score.

At least one of the data acquirer 131-1, the pre-processor 131-2, thelearning data selector 131-3, the model learner 131-4, and the modelevaluator 131-5 in the data learner 131 may be manufactured in the formof at least one hardware chip and may be installed in an electronicdevice. For example, at least one of the data acquirer 131-1, thepre-processor 131-2, the learning data selector 131-3, the model learner131-4, and the model evaluator 131-5 may be manufactured in the form ofa dedicated hardware chip for artificial intelligence (AI) or may bemanufactured as a part of an existing general-purpose processor (e.g., aCPU or an application processor) or a graphic dedicated processor (e.g.,a GPU) and may be installed in the aforementioned various electronicdevices.

The data acquirer 131-1, the pre-processor 131-2, the learning dataselector 131-3, the model learner 131-4, and the model evaluator 131-5may be installed in the electronic device 100 or may be installed inseparate electronic devices, respectively. For example, one or more ofthe data acquirer 131-1, the pre-processor 131-2, the learning dataselector 131-3, the model learner 131-4, and the model evaluator 131-5may be included in an electronic device and the others may be includedin a server.

At least one of the data acquirer 131-1, the pre-processor 131-2, thelearning data selector 131-3, the model learner 131-4, and the modelevaluator 131-5 may be implemented as a software module. When at leastone of the data acquirer 131-1, the pre-processor 131-2, the learningdata selector 131-3, the model learner 131-4, and the model evaluator131-5 is implemented as a software module (or a program module includingan instruction), the software module may be stored in a non-transitorycomputer readable medium (or media). In this case, at least one softwaremodule may be provided by an operating system (OS) or may be provided bya predetermined application. Alternatively, a portion of at least onesoftware module may be provided by an operating system (OS) and theremaining portion may be provided by a predetermined application.

FIG. 4A also provides a block diagram of the content controller 132according to an example embodiment of the present disclosure.

Referring to FIG. 4A, the content controller 132 according to someexample embodiments of the present disclosure may include a dataacquirer 132-1, a pre-processor 132-2, a recognition data selector132-3, a recognition result provider 132-4, and a model updater 132-5.However, the present disclosure is not limited thereto. According tovarious example embodiments of the present disclosure, a data classifier(not shown) may include some of the aforementioned components of thecontent controller 132. For example, the content controller 132 mayinclude only the data acquirer 132-1 and the recognition result provider132-4. According to various example embodiments of the presentdisclosure, the content controller 132 may further include othercomponents in addition to the aforementioned components.

The content controller 132 may estimate whether a content feature isincluded in content according to a reference for determining whether auser model corresponds to a learned content feature according toresponse characteristics of a user's eye(s) using at least one usermodel.

The data acquirer 132-1 may acquire input data required to estimate thecontent feature according to response characteristics of a user'seye(s). In this case, the data acquirer 132-1 may acquire video data.The video data may be stored in, for example, the memory 140 of theelectronic device 100 or may be data received from another electronicdevice (e.g., a smartphone, a tablet PC, and a server that haveestablished a communication relationship with the electronic device 100to transmit and receive content).

The pre-processor 132-2 may pre-process the acquired data to use theacquired input data to estimate a content feature. The pre-processor132-2 may process the acquired data in a predetermined format to use theacquired learning data when the recognition result provider 132-4 thatwill be described below estimates a content feature of an image.

For example, the pre-processor 132-2 may remove noise of the video dataacquired by the data acquirer 132-1 or process the video data to selectsignificant data.

The recognition data selector 132-3 may select data required forlearning from the pre-processed input data. The selected data may beprovided to the recognition result provider 132-4. To estimate whether acontent feature is included in an image(s), the recognition dataselector 132-3 may select all or some of the pre-processed data.

The recognition result provider 132-4 may apply the selected input datato a user model to estimate whether a content feature is included in animage. The recognition result provider 132-4 may use the data selectedby the recognition data selector 132-3 as an input value to apply theselected data to the user model. The recognition result provider 132-4may estimate, for example, whether a content feature is included in animage(s) based on a user model corresponding to a predeterminedcondition among at least one user model. A method of estimating acontent feature using input data by the recognition result provider132-4 will be described in more detail with reference to FIG. 6.

The model updater 132-5 may update a user model based on evaluation of arecognition result provided by the recognition result provider 132-4.For example, the model updater 132-5 may provide an estimation result ofa content feature included in an image provided by the recognitionresult provider 132-4 to the model learner 131-4 and, thus, the modellearner 131-4 may update the user model.

At least one of the data acquirer 132-1, the pre-processor 132-2, therecognition data selector 132-3, the recognition result provider 132-4,and the model updater 132-5 in the content controller 132 may bemanufactured in the form of at least one hardware chip and may beinstalled in the electronic device 100. For example, at least one of thedata acquirer 132-1, the pre-processor 132-2, the recognition dataselector 132-3, the recognition result provider 132-4, and the modelupdater 132-5 may be manufactured in the form of a dedicated hardwarechip for artificial intelligence (AI) or may be manufactured as a partof an existing general-purpose processor (e.g., a CPU or an applicationprocessor) or a graphic dedicated processor (e.g., a GPU) and may beinstalled in the aforementioned various electronic devices.

The data acquirer 132-1, the pre-processor 132-2, the recognition dataselector 132-3, the recognition result provider 132-4, and the modelupdater 132-5 may be installed in one electronic device or may beinstalled in separate electronic devices, respectively. For example, oneor more of the data acquirer 132-1, the pre-processor 132-2, therecognition data selector 132-3, the recognition result provider 132-4,and the model updater 132-5 may be included in the electronic device andthe others may be included in a server.

At least one of the data acquirer 132-1, the pre-processor 132-2, therecognition data selector 132-3, the recognition result provider 132-4,and the model updater 132-5 may be implemented as a software module.When at least one of the data acquirer 132-1, the pre-processor 132-2,the recognition data selector 132-3, the recognition result provider132-4, and the model updater 132-5 is implemented as a software module(or a program module including an instruction), the software module maybe stored in a non-transitory computer readable medium (or media). Inthis case, at least one software module may be provided by an operatingsystem (OS) or may be provided by a predetermined application.Alternatively, a portion of at least one software module may be providedby an operating system (OS) and the remaining portion may be provided bya predetermined application.

FIG. 4B is a diagram for explanation of an example in which anelectronic device and a server are operatively associated with eachother to learn and recognize data according to an example embodiment ofthe present disclosure.

Referring to FIG. 4B, a server 2000 may construct a user model forestimating of a content feature included in an image and a diseaseprediction model for estimating an abnormal state (e.g., a diseasestate) of an eye(s). The electronic device 100 may estimate a contentfeature included in content using the user model constructed by theserver 2000 and may estimate an abnormal state of a user eye using thedisease prediction model.

In this case, a data learner 2100 of the server 2000 may perform afunction of the data learner 131 shown in FIG. 4A. That is, a dataacquirer 2100-1, a pre-processor 2100-2, a learning data selector2100-3, a model learner 2100-4, and a model evaluator 2100-5 included inthe data learner 2100 of the server 2000 may correspond to the dataacquirer 131-1, the pre-processor 131-2, the learning data selector131-3, the model learner 131-4, and the model evaluator 131-5 shown inFIG. 4A.

The data learner 2100 of the server 2000 may learn responsecharacteristics of a user's eye(s) with respect to a content feature toestimate a content feature included in a video image. The data learner2100 may learn whether a user expresses specific responsecharacteristics with respect to a specific content feature and mayconstruct a user model (e.g., the user model 300 of FIG. 3) for storinga content feature according to response characteristics of a user eye.

The data learner 2100 of the server 2000 may learn a disease typeaccording to a state of an eye with a disease related to an eye. Thedata learner 2100 may learn whether an eye(s) is abnormal (e.g., adisease type) according to a change in an eye state during apredetermined time period to construct a disease prediction model.

The recognition result provider 132-4 of the electronic device 100 mayapply data selected by the recognition data selector 132-3 to the usermodel or the disease prediction model generated by the server 2000 toestimate a content feature or to estimate an abnormal state related to auser's eye(s).

For example, the recognition result provider 132-4 may transmit theinput data selected by the recognition data selector 132-3 to the server2000 and may apply the input data received by the server 2000 to theuser model or the disease prediction model to make a request forestimation of a content feature or to make a request for estimation ofan abnormal state related to an eye(s). The recognition result provider132-4 may receive information corresponding to the content feature orthe abnormal state related to an eye(s), estimated by the server 2000,from the server 2000.

FIG. 5 is a diagram for explanation of a method of constructing a usermodel by the model learner 131-4 according to an example embodiment ofthe present disclosure. The model learner 131-4 may include a responsecharacteristics detector 530 and a content feature detector 540.

First, the response characteristics detector 530 may acquire image dataincluding a user's eye(s) as learning data. In this case, the responsecharacteristics detector 530 may acquire the image data including auser's eye(s), pre-processed in a predetermined format in real time, butthis is merely an example embodiment and the response characteristicsdetector 530 may acquire only an image data including a user's eye(s)that expresses a feature response.

Since sizes and shapes of eyes are different for respective users, theresponse characteristics detector 530 may detect a change in shape andsize of a user's eye(s) using a user eye database (DB) 510 for storinginformation on eyes of a user who wears the electronic device 100 and aresponse characteristics DB 520 for storing information on a responsecharacteristics DB 520 of a general user's eye(s). For example, theresponse characteristics detector 530 may detect a change in an eyeshape (e.g., frown), eye blink, eye closure, pupil dilation, etc. asresponse characteristics of a user's eye(s).

The content feature detector 540 may acquire learning content. Thecontent feature detector 540 may analyze a content frame within apredetermined content section (e.g., a content section of about onesecond from a start point at which response characteristics of a usereye is detected) from a time point at which response characteristics ofa user's eye(s) are analyzed to detect a content feature. In this case,the content feature detector 540 may detect an object included in acontent frame, object, color change, brightness change, and so on as acontent feature.

When the content feature detector 540 detects a content feature, thecontent feature detector 540 may output response characteristics of auser's eye(s) and content feature corresponding thereto to the usermodel 300. When the content feature detector 540 does not detect acontent feature, the response characteristics detector 530 may determinethat response characteristics are not a response to content and mayremove the detected response characteristics.

The response characteristics detector 530 and the content featuredetector 540 may repeatedly perform the aforementioned method to learncontent features according to response characteristics of a user'seye(s).

The user model 300 may match the content feature and the responsecharacteristics of the user's eye(s) detected using the aforementionedmethod. For example, when a user frequently closes his/her eyes in ascene in which blood or a knife is displayed, the user model 300 maystore “eye closure” as response characteristics of a user's eye(s) andmay store “blood and knife” as the corresponding content feature. Asanother example, when a user frowns the eyes in scene transition with abrightness change of 100 or more (e.g., increase from 13 to 113), theuser model 300 may store “eye frown” as response characteristics of auser eye and may store “increase of brightness of 100” as thecorresponding content feature. As another example, when a user blinkshis/her eye(s) in an overall red scene, the user model 300 may store“eye blink” as response characteristics of a user's eye(s) and may store“red scene” as the corresponding content feature. As the aforementionedexamples, the user model 300 may store the following table.

TABLE 1 Response characteristics of user eye(s) content feature Eyeclosure Blood and knife Eye blink Increase of brightness of 100 Changein eye shape(frown) Red scene appearance

The user model 300 may update the content stored according to theresponse characteristics and the content feature that are detected bythe response characteristics detector 530 and the content featuredetector 540. For example, the user model 300 may delete the pre-storedcontent or may additionally store new content based on the resultdetected by the response characteristics detector 530 and the contentfeature detector 540. That is, the electronic device 100 maycontinuously learn a content feature according to responsecharacteristics of a user's eye(s) to construct a user model that isoptimized to a user.

The user model 300 may match and store response characteristics of auser's eye(s) and a content feature corresponding thereto and may alsostore various pieces of information such as user emotion information(e.g., negative emotion, positive emotion, fear, hatred, andpreference), a detection time point, and a number of times of detection.

A user model corrector 550 may correct the user model 300 according to auser input. For example, the user model corrector 550 may generate auser interface (UI) for adding/deleting response characteristics of auser's eye(s) and a content feature and may correct the responsecharacteristics of the user's eye(s) and the content feature accordingto a user input that is input through the UI. For example, as shown inFIG. 7A, the user model corrector 550 may display a UI 701 foradding/deleting the response characteristics of the user's eye(s) and,as shown in FIG. 7B, the user model corrector 550 may display a UI 702for designating a fear object of a user from a content feature.

In the aforementioned embodiment, a visual content feature and responsecharacteristics of a user's eye(s) are matched and stored but this ismerely an example embodiment and other (non-visual) content features(e.g., curse, bomb noise, and noise from Styrofoam) and responsecharacteristics of a user's eye(s) may be matched and stored.

FIG. 6 is a diagram for use in explaining a method of processing contentby the recognition result provider 132-4 of the content controller 132using the user model 300 according to an example embodiment of thepresent disclosure. According to an example embodiment of the presentdisclosure, the recognition result provider 132-4 may include a userrecognizer 610, a content feature recognizer 620, and a contentreproduction controller 630.

First, the user recognizer 610 may recognize a user who wears theelectronic device 100. In this case, the user recognizer 610 mayphotograph a user's iris using the image capture device 120 to recognizethe user, but this is merely an example embodiment and the user may berecognized using other methods and techniques (e.g., fingerprint, apassword, and voice recognition).

When a user is recognized, the content feature recognizer 620 mayanalyze whether a content feature stored in the user model 300 ispresent in reproduced content. In detail, the content feature recognizer620 may pre-analyze a content frame of a preceding content section froma currently reproduced content frame during reproduction of content toanalyze whether a content feature stored in the user model 300 ispresent. That is, the content feature recognizer 620 may determinewhether a specific object is detected, whether a specific color changeis detected, and whether a specific brightness change is detected, in acontent frame of a preceding content section of a currently reproducedcontent frame during reproduction of content.

When a content feature stored in the user model 300 is detected by thecontent feature recognizer 620, the content reproduction controller 630may process content based on a control model 640. In this case, thecontrol model 640 may match and store response characteristics of auser's eye(s) stored in the user model 300 and a method of processingcontent according to a content feature.

According to an example embodiment of the present disclosure, when adetected content feature is a specific object and responsecharacteristics with respect to the specific object are a negativeresponse (e.g., eye closure), the content reproduction controller 630may perform filtering processing on the object included in the contentor may perform smoothing processing using the control model 640. Forexample, as shown in FIG. 8A, when a detected content feature is a knife801 and response characteristics with respect to a knife are eyeclosure, the content reproduction controller 630 may perform filteringprocess on a knife region 802 of a content frame, as shown in FIG. 8B.

According to another example embodiment of the present disclosure, whena detected content feature is a specific brightness change and a userresponse to the specific brightness change is negative (e.g., eyefrown), the content reproduction controller 630 may adjust a changeamount of the specific brightness change. For example, when the detectedcontent feature is an increase of brightness of 100 and responsecharacteristics with respect to an increase of brightness of 100 is eyefrown, the content reproduction controller 630 may reduce a brightnessincrease value to 70 from 100. In this case, the content reproductioncontroller 630 may process an image to reduce the brightness increasevalue but this is merely an example embodiment and the contentreproduction controller 630 may control a backlight of the display 110to reduce the brightness increase value.

According to another example embodiment of the present disclosure, whenthe detected content feature is a specific color change and a userresponse to the specific color change is negative (e.g., eye blink), thecontent reproduction controller 630 may adjust a saturation value of aspecific color change. For example, when the detected content feature isa red scene and response characteristics with respect to appearance of ared scene are eye blink, the content reproduction controller 630 mayadjust the saturation value to display an orange scene instead of thered scene.

According to the aforementioned example embodiment, the detected contentfeature is a visual content feature, but this is merely an exampleembodiment and an acoustic content feature in addition to oralternatively to the visual content feature may also be included in thetechnological idea of the present disclosure. For example, when “curse”as a content feature and “eye frown” as response characteristicscorresponding thereto are stored in the user model 300, curse isdetected from reproduced content, the content reproduction controller630 may process curse on mute and may reproduce content.

In the aforementioned example embodiment, the detected responsecharacteristics of a user's eye(s) are negative, but this is merely anexample embodiment and, when response characteristics of a user eye arepositive, the positive response characteristics of the user eye and aresponse characteristics corresponding thereto may be matched and storedin the user model 300. For example, when a user pupil dilates in a scenein which a puppy is displayed, the user model 300 may store “puppy” as acontent feature and “pupil dilation” as response characteristicscorresponding thereto. In this case, when a user inputs a command forgenerating a content list in the future, the processor 130 may acquire akeyword (e.g., puppy) from a content feature corresponding to thepositive response characteristics and may generate a content listincluding the content feature based on a keyword.

The processor 130 may continuously photograph a user's eye(s) to checkdisease information (or health information) of a user eye and maycontrol the display 110 to provide the checked disease information orhealth-related information.

For example, when a user views a video image using the electronic device100, the processor 130 may continuously photograph a user's eye(s) tocheck for an abnormal state of the eye and may provide disease-relatedinformation or health-related information.

This will be described in more detail with reference to FIGS. 9 and 10.First, to check disease-related information of a user's eye(s), theprocessor 130 may include a user recognizer 910, a user eye imageacquirer 920, a state detector 930, a disease determiner 940, and aguider 950.

The user recognizer 910 may recognize a user based on the captured imageof a user's eye(s). In particular, the user recognizer 910 may recognizea user via iris recognition, but this is merely an example embodimentand the user recognizer 910 may recognize a user using other methods.

The user eye image acquirer 920 may acquire image data including auser's eye(s) photographed by the image capture device 120. In thiscase, the user eye image acquirer 920 may acquire the image dataincluding the user's eye(s), but this is merely an example embodimentand only information on a partial region (e.g., an eye or a pupil) ofthe image data may be acquired.

The state detector 930 may detect a state of the user's eye(s) includedin the image data. In this case, the state detector 930 may detect achange in a size, shape, color, etc. of the user's eye(s) based on auser eye DB 960 that stores a preceding image(s) of a user's eye(s).

According to an example embodiment of the present disclosure, a datalearner (e.g., the data learner 2100 of FIG. 4B) may generate diseaseprediction model 970.

To generate the disease prediction model 970, the data learner (e.g.,the data learner 2100 of FIG. 4B) may learn a type of a diseaseaccording to a state of an eye(s) having a disease related to an eye(s).In this case, the data learner (e.g., the data learner 2100 of FIG. 4B)for generating the disease prediction model 970 may learn whether aneye(s) is abnormal (e.g., a disease type) according to a change in aneye state during a predetermined time period to construct the diseaseprediction model 970.

According to an example embodiment of the present disclosure, thedisease prediction model 970 may be stored in the server 2000 locatedoutside the electronic device 100. According to various exampleembodiments of the present disclosure, the electronic device 100 mayreceive at least a portion of the disease prediction model 970 from theserver 2000 and may store the received portion in the memory 140, etc.of the electronic device 100.

The disease determiner 940 may determine whether a disease occursaccording to a state of the user's eye(s) detected by the state detector930 based on the disease prediction model 970. For example, when thestate detector 930 detects that a user's eye(s) is inflamed, the diseasedeterminer 940 may determine that iritis occurs.

As another example, when the state detector 930 detects that abnormalityoccurs in pupil dilation/contraction or response speed is lowered, thedisease determiner 940 may determine that Horner syndrome or holmesadie's tonic pupil occurs. As another example, when the state detector930 detects that pupil movement is abnormal based on head movement, thedisease determiner 940 may determine that otolithiasis occurs.

As another example, the disease determiner 940 may determine whetherNystagmus occurs according to eyeball movement. As another example, thedisease determiner 940 may detect whether optic nerve is damaged orglaucoma occurs according to whether visual field defect occurs. In thiscase, the disease determiner 940 may track pupil movement during use ofvirtual reality (VR) content to generate a visual field map and maydetermine whether visual field defect occurs through the generatedvisual field map.

The disease determiner 940 may also determine a disease occurrenceprediction time and symptom severity based on a size change, a shapechange, a color change, and response speed of an eye (or a pupil)detected by the state detector 930.

When the disease determiner 940 determines that a user's eye(s) has adisease, the guider 950 may provide a guidance message includingdisease-related information on the user's eye(s). For example, as shownin FIG. 10, the guider 950 may provide a guidance message 1010 such as“There is a high possibility of eye disease. Visit the hospital soon.”.In this case, the guidance message may include a disease type, diseaseoccurrence prediction time, disease severity, and so on as well aswhether a disease is predicted.

According to an example embodiment of the present disclosure, theelectronic device 100 may display a guidance message and/or anotherelectronic device (e.g., a smartphone and a tablet PC) that communicateswith the electronic device 100 may display the guidance message.

As such, according to various example embodiments of the presentdisclosure, a user may also check a health state of an eye while viewingcontent such as video images using the electronic device 100. Inparticular, with regard to estimation of an abnormal state related to aneye using the disease prediction model according to the presentdisclosure, the electronic device 100 may estimate an abnormal state ofan eye according to a change in an eye state for a predetermined time(e.g., a time of 1 hour or more) to increase the accuracy of examinationand may also reduce a user inconvenience of preparing a separate timefor eye examination.

FIG. 11 is a flowchart of a method of controlling an electronic deviceaccording to an example embodiment of the present disclosure.

First, the electronic device 100 may learn a content feature accordingto response characteristics of an eye of a user who wears the electronicdevice 100 to construct a user model (S1110). In detail, the electronicdevice 100 may capture an image including the eye(s) of the user whowears an electronic device during reproduction of learning content and,upon detecting predetermined response characteristics of the user'seye(s) included in the image, the electronic device 100 may acquire acontent feature included in a content frame within a predeterminedcontent section from a time point at which the predetermined responsecharacteristics are detected and may learn the predetermined responsecharacteristics and the content feature to construct a user model.

The electronic device 100 may reproduce content (S1120).

The electronic device 100 may detect a content feature stored in theuser model from the reproduced content (S1130). In detail, theelectronic device 100 may analyze a content frame of a preceding contentsection of a currently reproduced content frame during reproduction ofcontent to determine whether a content feature stored in the user modelis present.

Upon detecting a content feature included in the user model (S1130-Y),the electronic device 100 may process reproduced content based onresponse characteristics of a user's eye(s) corresponding to thedetected content feature (S1140). In detail, the electronic device 100may process an image of a content frame including the detected contentfeature based on the detected content feature and responsecharacteristics corresponding to the detected content feature. Forexample, the electronic device 100 may perform smoothing processing andfiltering processing on a content frame including a content feature ormay adjust a brightness change value or a saturation value.

According to the aforementioned various example embodiments of thepresent disclosure, an image of a content feature expressed as a usernegative response may be processed and, thus, the user may view theimage more conveniently without adverse stimulation.

According to the aforementioned example embodiments of the presentdisclosure, the electronic device 100 constructs a user model andprocesses content using the constructed user model, but these are merelyexample embodiments and, thus, as shown in FIG. 12, a portable terminal1200 connected to the electronic device 100 may construct the user modeland may process content using the constructed user model.

In detail, while the portable terminal 1200 transmits learning contentto the electronic device 100 and the electronic device 100 reproducesthe learning content, the electronic device 100 may photograph a user'seye(s) and may transmit the captured image data to the portable terminal1200. In this case, upon detecting predetermined responsecharacteristics of a user's eye(s), the electronic device 100 maytransmit the captured image data to the portable terminal 1200.

The portable terminal 1200 may construct a user model for storingresponse characteristics of a user's eye(s) and a content featurecorresponding thereto based on the learning content and the capturedimage data.

Upon transmitting content including a content feature included in theuser model to the electronic device 100, the portable terminal 1200 mayprocess an image of content based on a content feature included in theuser model and response characteristics of a corresponding user's eye(s)and, then, may transmit the image-processed content to the electronicdevice 100.

FIG. 13 is a flowchart used for explaining of a case of estimating acontent feature when an electronic device includes a first processor anda second processor according to an example embodiment of the presentdisclosure.

Referring to FIG. 13, the electronic device 100 may include a firstprocessor 130 a and a second processor 130 b.

The first processor 130 a may control execution of at least one ofapplication installed in the electronic device 100 and may performgraphic processing on an image (e.g., a live view image, a capturedimage, and a video image) acquired by the electronic device 100. Thefirst processor 130 a may be implemented in the form of a system on chip(SoC) formed by integrating functions of a central processing unit(CPU), a graphic processing unit (GPU), a communication chip, a sensor,and so on. The first processor 130 a may also be referred to as anapplication processor (AP) in the present disclosure.

The second processor 130 b may estimate an interest region of an imageusing a data recognition model.

The second processor 130 b may be manufactured in the form of adedicated hardware chip for artificial intelligence (AI) for performinga function of estimating an interest region using the data recognitionmodel. According to various example embodiments of the presentdisclosure, in the case of a data recognition model using visualunderstanding as a core technology, the dedicated hardware chip forartificial intelligence (AI) may include a GPU.

The electronic device 100 may further include a third processor, afourth processor, and so on, for performing the same function(s) as thesecond processor 130 b.

According to various example embodiments of the present disclosure, afunction performed by the first processor 130 a may be stored in thememory 140 and may correspond to an application for performing variousfunctions, and a function performed by the second processor 130 b maycorrespond to an OS of the electronic device 100.

For example, a camera application may generate a live view image and maydetermine a data recognition model corresponding to a predeterminedcondition. The camera application may transmit information related to aninterest region estimation request and a data recognition modeldetermined with respect to an OS and/or a server positioned outside(external to) the electronic device 100.

The OS and/or the external server may estimate an interest region usingthe included data recognition model.

According to an example embodiment of the present disclosure, the firstprocessor 130 a may reproduce content (S1310).

For example, the first processor 130 a may reproduce a video image(s)stored in a memory or may receive streaming video data from an externalserver and may reproduce the streaming the video data.

The first processor 130 a may transmit a frame that precedes a frame tobe currently reproduced by a predetermined time, to the second processor130 b (S1320).

The second processor 130 b may apply the received frame to the usermodel to estimate a content feature included in a frame (S1330).

When a content feature is estimated, the second processor 130 b maytransmit the estimated content feature to the first processor 130 a(S1340).

The first processor 130 a may determine a content processing methodcorresponding to the estimated content feature and may apply the contentprocessing method to content. For example, the first processor 130 a mayperform smoothing processing and filtering processing on a frameincluding a content feature or may adjust a brightness value or asaturation value (S1350).

The first processor may reproduce the content to which the contentprocessing method is applied (S1360).

FIG. 14 is a flowchart used for explaining a case of estimating acontent feature using a server by an electronic device according to anexample embodiment of the present disclosure.

As described above with reference to FIG. 4B, the server 2000 accordingto an example embodiment of the present disclosure may include a usermodel.

According to an example embodiment of the present disclosure, theelectronic device 100 may reproduce content (S1410).

For example, the electronic device 100 may reproduce a video image(s)stored in a memory or may receiving streaming video data from anexternal server and may reproduce the streaming video data.

The electronic device 100 may transmit a frame that precedes a frame tobe currently reproduced by a predetermined time, to the server 2000(S1420).

The server 2000 may apply the received frame to the user model toestimate a content feature included in the frame (S1430).

Upon estimating the content feature, the server 2000 may transmit theestimated content feature to the electronic device 100 (S1440).

The electronic device 100 may determine a content processing methodcorresponding to the estimated content feature and may apply the contentprocessing method to content. For example, the electronic device 100 mayperform smoothing processing or filtering processing on a frame includedin a content feature or may adjust a brightness value or a saturationvalue (S1450).

The electronic device 100 may reproduce the content to which the contentprocessing method is applied (S1460).

FIG. 15 is a flowchart used for explaining a case of estimating anabnormal state related to an eye(s) using a server by an electronicdevice according to an example embodiment of the present disclosure.

According to an example embodiment of the present disclosure, the server2000 may include a disease prediction model.

According to an example embodiment of the present disclosure, theelectronic device 100 may reproduce content (S1510).

For example, the electronic device 100 may reproduce a video image(s)stored in a memory or may receive video data from an external server andmay reproduce the video data.

The electronic device 100 may photograph a user's eye(s) for apredetermined time or more (S1520). For example, while a user views avideo image, the electronic device 100 may photograph a user's eye(s)who views the video image, for about one hour. In addition, according tovarious example embodiments of the present disclosure, while a userviews a video image, the electronic device 100 may photograph a user'seye(s) that views the video image for about 5 seconds and may repeatedlyre-perform photographing for about five seconds after one minuteintervals.

The electronic device 100 may transmit the photographed eye images tothe server 2000 (S1530).

The server 2000 may apply the received eye images to the diseaseprediction model to estimate an abnormal state related to an eye(s)(S1540). For example, when the server 2000 applies an image(s) in whicha surrounding area of a user's eye(s) becomes red to a diseaseprediction model, the disease prediction model may estimate that a usereye has iritis.

The server 2000 may transmit the estimated disease-related content tothe electronic device 100 (S1550).

The electronic device 100 may displayed the received disease-relatedcontent (S1560). According to various example embodiments of the presentdisclosure, another electronic device (e.g., a smartphone and a tabletPC) that communicates with the electronic device 100 may display thereceived disease-related content.

The aforementioned methods can include a computer readable mediumincluding program commands for executing operations implemented usingvarious computers. The computer readable medium can store programcommands, data files, data structures or combinations thereof. Theprogram commands recorded in the medium may be specially designed andconfigured for the present disclosure or be known to those skilled inthe field of computer software. Examples of a computer readablerecording medium include magnetic media such as hard discs, floppy discsand magnetic tapes, optical media such as CD-ROMs and DVDs,magneto-optical media such as floptical discs, or hardware devices suchas ROMs, RAMs and flash memories, which are specially configured tostore and execute program commands. Examples of the program commandsinclude machine language code created by a compiler and high-levellanguage code executable by a computer using an interpreter and thelike. The hardware device may be configured to operate as one or moresoftware modules to perform an operation according to the presentdisclosure, or vice versa.

The example embodiments may be implemented in software program includingcommands stored in a computer-readable storage medium (or media).

The computer may be a device that is capable of calling a command storedin a storage medium and performing an operation according to the exampleembodiments of the present disclosure according to the called commandand may include electronic devices according to the example embodimentsof the present disclosure.

The computer-readable storage medium may be provided in the form of anon-transitory storage medium. Here, the term ‘non-transitory’ mayindicate that a storage medium does not include a signal and istangible.

The control method according to the example embodiments of the presentdisclosure may be included in a computer program product. The computerprogram product may be traded between a seller and purchaser.

The computer program product may include an software program and acomputer readable medium (or media) having recorded thereon the softwareprogram. For example, the computer program product may include asoftware program type of product (e.g., a downloadable app) that iselectronically distributed through a manufacturer or an E-market (e.g.,a Google player store and an app store) of the electronic device. Forelectronic distribution, at least a portion of a software program may bestored in a storage medium or may be temporally generated. In this case,the storage medium may be a server of a manufacturer, a server of anE-market, or a storage medium of a relay server for temporally storing asoftware program.

The computer program product may include a storage medium of a server ora storage medium of an electronic device in a system including a serverand an electronic device. In addition, when a third device (e.g., asmartphone) communication-connected to a server or an electronic deviceis present, the computer program product may include a storage medium ofthe third device. The computer program product may be transmitted to anelectronic device or a third device from a server or may include asoftware program transmitted to the electronic device from the thirddevice.

In this case, one of a server, an electronic device, and a third devicemay execute the computer program product to perform the methodsaccording to the aforementioned example embodiments of the presentdisclosure. Alternatively, two or more of the electronic device and thethird device may execute the computer program product to distribute andimplement the method according to the aforementioned example embodimentsof the present disclosure.

For example, a server (e.g., a cloud server or an artificialintelligence (AI) server) may execute the computer program productstored in the server and an electronic device that iscommunication-connected to the server may be controlled to perform themethods according to the example embodiments of the present disclosure.

As another example, the third device may execute the computer programproduct and the electronic device that is communication-connected to thethird device may be controlled to perform the methods according to theexample embodiments of the present disclosure. When the third deviceexecutes the computer program product, the third device may download thecomputer program product from the server and may execute the downloadedcomputer program product. The third device may execute the computerprogram product provided in a preloaded state to perform the methodaccording to the example embodiments of the present disclosure.

The foregoing example embodiments and advantages are merely examples andare not to be construed as limiting the present disclosure. The presentteaching can be readily applied to other types of apparatuses. Also, thedescription of the example embodiments of the present disclosure isintended to be illustrative, and not to limit the scope of the claims,and many alternatives, modifications, and variations will be apparent tothose skilled in the art.

What is claimed is:
 1. A method of controlling an electronic device wornby a user to provide images, the method comprising: constructing, bytraining, a user model in which one or more content features are eachstored in association with a corresponding response characteristic of aneye of a user wearing the electronic device; detecting content featuresin content during reproduction of the content for perception by theuser; in response to detecting, by the content feature detecting, of acontent feature reflected in the user model, identifying the responsecharacteristic of the eye of the user corresponding to the detectedcontent feature, using the user model; and controlling the reproductionof the content for perception by the user according to a contentreproduction control method based on the identified responsecharacteristic of the eye of the user, wherein the detecting of contentfeatures in content comprises analyzing a preceding content section of acurrently reproduced content during the reproduction for perception bythe user of the currently reproduced content to analyze whether acontent feature reflected in the user model is present in the precedingcontent section, and wherein the content reproduction control method isdetermined based on a control model in which response characteristics ofthe eye of the user are stored in association with content reproductioncontrol methods.
 2. The method as claimed in claim 1, wherein theconstructing of the user model comprises: capturing images including theeye of the user wearing the electronic device during reproduction oftraining content; in response to detecting a predetermined responsecharacteristic of the eye of the user included in one or more of thecaptured images, acquiring a content feature included in a content framewithin a content section of the training content relative to a timepoint at which the predetermined response characteristic is detected;and constructing the user model by associating the predeterminedresponse characteristic and the acquired content feature.
 3. The methodas claimed in claim 2, wherein the acquired content feature comprises atleast one of an object included in the content frame within the contentsection, a brightness change of the content frame, or a color change ofthe content frame.
 4. The method as claimed in claim 1, wherein thecontrolling of the reproduction of the content comprises: in response todetecting, in the content, a first content feature reflected in the usermodel, controlling the reproduction of the content by processing animage of a content frame comprising the first content feature accordingto a content reproduction control method determined based on theresponse characteristic of the eye of the user identified ascorresponding to the first content feature.
 5. The method as claimed inclaim 4, wherein the controlling of the reproduction of the contentcomprises: based on the first content feature being a specific objectand the identified response characteristic in the user model of thespecific object corresponding to a negative response, performing acontent reproduction control method for filtering processing orsmoothing processing on the object included in the content.
 6. Themethod as claimed in claim 4, wherein the controlling of thereproduction of the content comprises: based on the first contentfeature being a specific brightness change and the identified responsecharacteristic in the user model to the specific brightness changecorresponding to a negative response, performing a content reproductioncontrol method for adjusting a change amount of the specific brightnesschange.
 7. The method as claimed in claim 4, wherein the controlling ofthe reproduction of the content comprises: based on the first contentfeature being a specific color change and the identified responsecharacteristic in the user model to the specific color changecorresponding to a negative response, performing a content reproductioncontrol method for adjusting a saturation value of the specific colorchange.
 8. The method as claimed in claim 2, further comprising: basedon identifying that the response characteristic in the user model to aparticular content feature corresponds to a positive response, acquiringa keyword associated with the particular content feature; and providinga list comprising recommended content determined based on the acquiredkeyword.
 9. The method as claimed in claim 1, further comprising:cumulatively storing captured images including the eye of the user;analyzing the stored images to determine whether an abnormal state ofthe eye of the user is identified; and based on determining that anabnormal state of the eye of the user is identified, providinginformation relating to the abnormal state.
 10. An electronic deviceworn by a user to provide images, comprising: a display; a camera; amemory; and a processor configured to: construct, by training, a usermodel, for storage in the memory, in which one or more content featuresare each associated with a corresponding response characteristic of aneye of user wearing the electronic device; detect content features incontent during reproduction of the content for perception by the user;in response to detecting, by the content feature detecting, of a contentfeature reflected in the user model, identify the responsecharacteristic of the eye of the user corresponding to the detectedcontent feature, using the user model; and control reproduction of thecontent for perception by the user according to a content reproductioncontrol method based on the identified response characteristic of theeye of the user, wherein the detecting of content features in contentcomprises analyzing a preceding content section of a currentlyreproduced content during the reproduction for perception by the user ofthe currently reproduced content to analyze whether a content featurereflected in the user model is present in the preceding content section,and wherein the content reproduction control method is determined basedon a control model in which response characteristics of the eye of theuser are stored in association with content reproduction controlmethods.
 11. The electronic device as claimed in claim 10, wherein theprocessor is further configured to: control the camera to capture imagesincluding the eye of the user wearing the electronic device duringreproduction of training content; in response to detecting apredetermined response characteristic of the eye of the user included inone or more of the captured images, acquire a content feature includedin a content frame within a content section of the training contentrelative to a time point at which the predetermined responsecharacteristic is detected; and construct the user model by associatingthe predetermined response characteristic and the acquired contentfeature.
 12. The electronic device as claimed in claim 11, wherein theacquired content feature comprises at least one of an object included inthe content frame within the content section, a brightness change of thecontent frame, or a color change of the content frame.
 13. Theelectronic device as claimed in claim 10, wherein the processor isconfigured to, in response to detecting, in the content, a first contentfeature reflected in the user model, control the reproduction of thecontent by processing an image of a content frame comprising the firstcontent feature according to a content reproduction control methoddetermined based on the response characteristic of the eye of the useridentified as corresponding to the first content feature.
 14. Theelectronic device as claimed in claim 13, wherein the processor isconfigured to, based on the first content feature being a specificobject and the identified response characteristic in the user model ofthe specific object corresponding to a negative response, control thereproduction of the content by performing a content reproduction controlmethod for filtering processing or smoothing processing on the objectincluded in the content.
 15. The electronic device as claimed in claim13, wherein the processor is configured to, based on the first contentfeature being a specific brightness change and the identified responsecharacteristic in the user model to the specific brightness changecorresponding to a negative response, control the reproduction of thecontent by performing a content reproduction control method foradjusting a change amount of the specific brightness change.
 16. Theelectronic device as claimed in claim 13, wherein the processor isconfigured to, based on the first content feature being a specific colorchange and the identified response characteristic in the user model tothe specific color change corresponding to a negative response, controlthe reproduction of the content by performing a content reproductioncontrol method for adjusting a saturation value of the specific colorchange.
 17. The electronic device as claimed in claim 11, wherein theprocessor is configured to: based on identifying that the responsecharacteristic in the user model to a particular content featurecorresponds to a positive response, acquire a keyword associated withthe particular content feature; and provide a list comprising arecommended content determined based on the acquired keyword.
 18. Theelectronic device as claimed in claim 10, wherein the processor isconfigured to: cumulatively store captured images including the eye ofthe user; analyze the stored images to determine whether an abnormalstate of the eye of the user is identified; and based on determiningthat an abnormal state of the eye of the user is identified, provideinformation relating to the abnormal state.